Image Caption Generator Based On Deep Neural Networks

In this project, we systematically analyze a deep neural networks based image caption generation method. With an image as the input, the method can output an English sentence describing the content in the image. We analyze three components of the method: convolutional neural network (CNN), recurrent neural network (RNN) and sentence generation. By replacing the CNN part with three state-of-the-art architectures, we find the VGGNet performs best according to the BLEU score. We also propose a simplified version the Gated Recurrent Units (GRU) as a new recurrent layer, implementing by both MATLAB and C++ in Caffe. The simplified GRU achieves comparable result when it is compared with the long short-term memory (LSTM) method. But it has few parameters which saves memory and is faster in training. Finally, we generate multiple sentences using Beam Search. The experiments show that the modified method can generate captions comparable to the-state-of-the-art methods with less training memory.